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CROSS-REFERENCE TO RELATED APPLICATION 

This application is based on, and claims priority to, U.S. Provisional Application 
No. 60/251,085, filed on December 4, 2000, which is fully incorporated herein by 
reference. 

COPYRIGHT NOTICE 

A portion of the disclosure of this patent document contains material which is 

subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction by any one of the patent document or the patent disclosure, as it appears in 
the patent and Trademark Office patent file or records, but otherwise reserves all 
copyright rights whatsoever. 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates generally to systems and methods for building 
speech-based applications and architectures and, in particular, to server-side and 
client-side speech firameworks employing reusable dialog components based on 
VoiceXML (Voice extensible Markup Language). 

BACKGROUND OF THE INVENTION 

The computing world is evolving towards an era where billions of interconnected 
pervasive clients will communicate with powerful information servers. Indeed, this 
millennium will be characterized by the availabiUty of multiple information devices that 
make ubiquitous information access an accepted fact of life. This evolution towards 
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billions of pervasive devices being interconnected via the Internet, wireless networks or 
spontaneous networks (such as Bluetooth and Jini) will revolutionize the principles 
underlying man-machine interaction. In the near future, personal information devices 
will offer ubiquitous access, bringing with them the ability to create, manipulate and 
exchange any information anywhere and anytime using interaction modalities most suited 
to the an individuaFs current needs and abilities. Such devices will include fanailiar 
access devices such as conventional telephones, cell phones, smart phones, pocket 
organizers, PDAs and PCs, which vary widely in the interface peripherals they use to 
communicate with the user. 

The increasing availability of information, along with the rise in the 
computational power available to each user to manipulate this information, brings with it 
a concomitant need to increase the bandwidth of man-machine communication. The 
ability to access information via a multiplicity of appliances, each designed to suit the 
individual's specific needs and abilities at any given time, necessarily means that these 
interactions should exploit all available input and output (I/O) modalities to maximize the 
bandwidth of man-machine communication. Indeed, users will come to demand such 
multi-modal interaction in order to maximize their interaction with information devices in 
hands-free, eyes-free environments. 

VoiceXML is a markup language designed to facilitate the creation of speech 
applications such as IVR (Interactive Voice Response) applications. Compared to 
conventional IVR programming frameworks that employ proprietary scripts and 
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programming langxiages over proprietary/closed platforms, the VoiceXML standard 
provides a declarative programming framework based on XML (extensible Markup 
Language) and ECMAScript (see, e.g., the W3C XML specifications 
(www.w3.org/XML) and VoiceXML forum (www, voicexml.org)). VoiceXML is 
designed to run on web-like infrastructures of web servers and web application servers 
(i.e. the Voice browser). VoiceXML is a key component for providing a voice interface 
to Mobile e-business. Indeed, VoiceXML allows information to be accessed by voice 
through a regular phone or a mobile phone whenever it is difficult or not optimal to 
interact through a wireless GUI micro-browser. 

More importantly, VoiceXML is a key component to building multi-modal 
systems such as multi-modal and conversational user interfaces or mobile multi-modal 
browsers. Multi-modal e-business solutions exploit the fact that different interaction 
modes are more efficient for different user interactions. For example, depending on the 
interaction, talking may be easier than typing, whereas reading may be faster than 
listening. Multi-modal interfaces combine the use of multiple interaction modes, such as 
voice, keypad and display to improve the user interface to e-business. Advantageously, 
multi-modal browsers can rely on VoiceXML browsers and authoring to describe and 
render the voice interface. 

There are still key inhibitors to the deployment of compelling multi-modal 
e-business applications. Most arise out of the current infrastructure and device platforms. 
Indeed, the current networking infrastructure is not configured for providing seamless, 
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multi-modal access to infomiation. Indeed, although a plethora of infomiation can be 
accessed from servers over a communications network using an access device (e.g., 
personal information and corporate information available on private networks and public 
information accessible via a global computer network such as the Internet), the 
availability of such information may be limited by the modaUty of the client/access device 
or the platform-specific software applications with which the user is interacting to obtain 
such information. For instance, current wireless network infrastructure and handsets do 
not provide simultaneous voice and data access. Middleware, interfaces and protocols are 
needed to synchronize and manage the different channels. 

Currently, application authoring methodologies are being developed to provide 
means to develop rich multi-modal applications. It is anticipated that most multi-modal 
mobile deployment will rely on wireless PDAs that can overcome the above challenges 
by hosting a VoiceXML browser on the cUent (fat cUent configuration) or by relying on 
sequential or notification-based multi-modal scenarios, where the user switches 
connectivity when he or she wants to interact through another modality. 

Because of the inherent challenges of conversational engines (e.g., speech 
recognizer) that require data files (e.g., grammars), however, it is important to provide 
mechanisms that provide tools that hide this level of complexity. It is also important that 
such mechanisms and tools overcome some of the limitations imposed by VoiceXML 
(e.g. the VoiceXML execution model). Thus, while it is anticipated that voice (alone for 
multi-channel applications) and multi-modal will be key catalyst to wide adoption of 
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mobile e-business, it is believed that such wide spread adoption of such voice and 
multi-modal interfaces will remain challenging until tools for building applications using 
voice-based reusable dialog components are available to non-speech speciahsts. 

The VoiceXML Forum has submitted VoiceXML 1.0 to the W3C Voice Browser 
activity (see, e.g., W3C voice browser activity, www.w3.org/voice/). As part of its 
activities, the W3C Voice Browser working group has identified reusable dialog 
components as an item worth studying and it pubUshed a set of associated requirements 
(see, e.g., the W3C reusable dialog requirements for voice markup language 
(www.w3 .org/TR/reusable-dialog-reqs)). 

Accordingly, VoiceXML frameworks for reusable dialog components 
(server-centric and client-centric), which satisfy the published W3C reusable dialog 
component requirements while remaining compUant with the VoiceXML specifications, 
for example, would be a significant component for building voice interfaces and 
multi-modal appUcations that seamlessly operate across a plurality of channels. 

SUMMARY OF THE INVENTION 

The present invention relates generally to systems and methods for building 
speech-based applications and, in particular, to server-side and chent-side speech 
fi-ameworks employing reusable dialog components based on VoiceXML (Voice 
extensible Markup Language). VoiceXML reusable dialog components according to the 
present invention can be used for building a voice interface for use with multi-modal. 
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multi-chaimel and conversational applications that offer universal access to information 
anytime, from any location, using any pervasive computing device regardless of its I/O 
modality. 

In one aspect of the present invention, a method for autiioring a speech application 
comprises the steps of: 

creating one or more reusable VoiceXML dialog components; 

creating an associated parameter object for each of the reusable VoiceXML 
dialog components; and 

creating a VoiceXML document comprising code for invoking a reusable 
VoiceXML dialog component and code for configuring the invoked reusable VoiceXML 
dialog component using an associated parameter object. 

In another aspect, a speech application server comprises: 

a VoiceXML processor for parsing and rendering a VoiceXML document; and 

a library comprising one or more reusable VoiceXML dialog components that are 
accessible by the VoiceXML processor, wherein the VoiceXML document comprises 
code for invoking a reusable VoiceXML dialog component and code for configuring the 
invoked reusable VoiceXML dialog component using an associated parameter object. 

In a preferred embodiment, a client-side reusable dialog component framework is 
built within the VoiceXML specifications and utilizes <subdialog> elements to call 
reusable VoiceXML dialog components and associated ECMAScript parameter objects to 
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pass parameters, configuration and results. This solution is interpreted at the client side 
(VoiceXML browser). 

In another aspect of the present invention, a server-side speech application server 
comprises: 

a VoiceXML page generation engine for dynamically building a VoiceXML page; 

a first database comprising one or more server-side reusable VoiceXML dialog 
components that are accessible the VoiceXML page generation engine for generating an 
intermediate VoiceXML page; 

a second database comprising backend data that is accessible by the VoiceXML 
page generator to insert data in the intermediate VoiceXML page to generate a 
VoiceXML page that is served to a requesting chent. 

Preferably, a server-side framework for reusable dialog components is based on 
JSP (Java Server Pages) and beans that generate VoiceXML subdialogs. 

In another aspect of the present invention, the server-side and client-side 
frameworks for reusable VoiceXML dialog components can be combined to provide a 
desired solution. 

In another aspect of the present invention, reusable VoiceXML dialog components 
are "re-entrant" to allow the reusable VoiceXML dialog components to be initiated, 
interrupted, inspected, and/or resumed with a partially filled result object or state object. 
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These and other aspects, features, and advantages of the present invention will 
become apparent from the following detailed description of the preferred embodiments, 
which is to be read in connection with the accompanying drawings. 

DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram of a system and method for implementing reusable VoiceXML 
dialog components according to an embodiment of the present invention. 

Fig. 2 is a diagram of a system and method for implementing reusable VoiceXML 
dialog components according to another embodiment of the present invention. 

Fig. 3 is a diagram of a system and method for implementing reusable VoiceXML 
dialog components using JSP (JavaServer Pages) and beans according to an embodiment 
of the present invention. 

Fig. 4 is a diagram of a system and method for implementing reusable VoiceXML 
dialog components according to another embodiment of the present invention. 

Fig. 5 is a diagram illustrating an execution flow of an invoked subdialog 
according to current VoiceXML specifications. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

The present invention relates generally to systems and methods for building 
speech-based applications and, in particular, to server-side and client-side speech 
frameworks employing reusable dialog components based on VoiceXML (Voice 
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extensible Markup Language). VoiceXML reusable dialog components according to the 
present invention can be used for building speech interfaces for multi-modal, 
multi-channel and conversational applications that offer xmiversal access to information 
anytime, from any location, using any pervasive computing device regardless of its I/O 
modality. 

It is to be understood that the term "channel" used herein refers to a particular 
renderer, device, or a particular modality. Examples of different modalities/channels 
include speech such as VoiceXML, visual (GUI) such as HTML (hypertext markup 
language), restrained GUI such as WML (wireless markup language), CHTML (compact 
HTML), XHTML-MP (XHTML Mobile profile), and HDML (handheld device markup 
language) or any combination of such modalities. 

The term "multi-channel application" refers to an application that provides 
ubiquitous access through different channels (e.g., VoiceXML, HTML), one channel at a 
time. Multi-channel applications do not provide synchronization or coordination across 
the views of the different channels. 

The term "multi-modal" appUcation refers to multi-channel appUcations, wherein 
multiple channels are simultaneously available and synchronized. Furthermore, from a 
multi-channel point of view, multi-modahty can be considered another channel. 

Furthermore, the term "conversational" or "conversational computing" as used 
herein refers to seamless multi-modal dialog (iirformation exchanges) between user and 
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machine and between devices or platforms of varying modalities (I/O capabilities), 
regardless of the I/O capabilities of the access device/channel, preferably, using open, 
interoperable communication protocols and standards, as well as a conversational (or 
interaction-based) programming model that separates the apphcation data content (tier 3) 
and business logic (tier 2) from the user interaction and data model that the user 
manipulates. The term "conversational application" refers to an application that supports 
multi-modal, free flow interactions (e.g., mixed initiative dialogs) within the application 
and across independently developed applications, preferably using short tenn and long 
term context (including previous input and output) to disambiguate and understand the 
user's intention. Conversational application preferably utilize NLU (natural language 
understanding). 

The following detailed description of preferred embodiments is divided into the 
following sections for ease of reference: Section I below provides a general description of 
VoiceXML reusable components according to the present invention, as well as the need, 
motivation and advantages of implementing frameworks based on VoiceXML reusable 
dialog components; Section n describes preferred embodiments of VoiceXML dialog 
component frameworks according to the present invention; Section in describes preferred 
dialog object interfaces and behaviors to support mixed initiative across subdialogs, 
documents and modahties using VoiceXML reusable components; and Section IV 
enumerates preferred specification standards for VoiceXML dialog components that fall 
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within the W3C speech framework, as well as extensions for future platforms and 
standards. 

I. Motivation For Employing VoiceXML Reasable Dialog Components 

As noted above, specifications for reusable components are being developed 
within the W3C speech fi-amework. According to the W3C reusable dialog requirements, 
reusable dialog components provide prepackaged functionality "out-of-the-box" that 
enables developers to build appUcations by providing standard default settings and 
behavior. The reusable dialog components shield developers from having to worry about 
many of the intricacies associated with building a robust speech dialogue, e.g., confidence 
score interpretation, error recovery mechanisms, prompting, etc. This behavior can be 
customized by a developer, if desired, to provide application-specific prompts, 
vocabulary, retry settings, etc. 

Reusable dialog components are classified herein as 'task" types and **template" 
types. Task components are designed to obtain some piece or pieces of information (e.g., 
get a date). Although task components can be configured, they will operate as-is. 
Template components require configuration and need to be parameterized (e.g., select a 
menu item, wherein the menu list must be provided. 

The VoiceXML specifications identify the possibility of using declarative reusable 
VoiceXML pages to create a reusable library of dialogs shared among many applications. 
It has also been recognized that prepackaged dialogs designed to reduce the application 
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developers effort through appropriate abstraction can occur at many levels and can be 
implemented in a number of different ways. For example, parameterized dialogs can be 
implemented as markup elements with attributes and sub-elements, as scripts built of 
markup elements and variables (perhaps stored in a standard library of such dialogs), or 
as native, pre-compiled, or otherwise non-markup language objects or modules. 
Therefore, from an application developer's point of view, it is advantageous to provide 
prepackaged reusable dialog components and sample code that can be use as libraries or 
as sample code / templates to build more complex applications and reusable dialog 
modules or customize them. 

Further, it would be highly advantageous to provide dialog reusable components 
authored in VoiceXML. Indeed, currently, there are a variety of VoiceXML applications 
that are being used or developed in which VoiceXML reusable components would 
provide a mechanism for seamless and synchronized multi-modal interactions across a 
plurality of channels. Such speech applications include, for example, IVR-centric, 
server-centric, cUent-centric and embedded implementations. Further, reusable 
VoiceXML components (which are based on extensions of the VoiceXML standard) 
would enable development of mixed initiative speech applications, as well as integrated 
speech applications (VoiceXML) within existing web infrastructures (multi-channel, 
multi-modal and conversational applications). 
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The following sections describe frameworks according to the present invention for 
implementing reusable dialog components, subdialogs, and beans, which are built within 
VoiceXML specifications. 

IL Frameworks For Reusable VoiceXML Dialog Components 

In one embodiment of the present invention, reusable VoiceXML dialog 
components are built using VoiceXML <subdialog> elements and using associated 
ECMAScript parameter objects to pass parameters, configuration and results. This 
framework comprises a client-centric approach (e.g., VoiceXML browser) using-side 
reusable dialog components. The present inventions provides various frameworks for 
reusable dialog components built within the VoiceXML specifications. 

In another embodiment, server-centric framework for implementing reusable 
VoiceXML components, wherein VoiceXML pages are dynamically generated to provide, 
e.g., dynamic manipulation or prompts, dynamic grammar compilation and dynamic 
access to data sources. In one preferred embodiment, a server-centric framework is based 
on JSP (Java Server Pages) and beans that generate VoiceXML subdialogs. In other 
embodiments of the present invention, a reusable VoiceXML component framework can 
be built using a combination of client-side and server-side reusable dialog components. 

It is to be understood that the term "cUent-side" or "cUent-centric" processing as 
used herein refers to processing that occurs directly at the level of the presentation layer 
which, in the present invention, comprise a VoiceXML browser. This does not memi 
that the VoiceXML browser is actually physically located on the chent. The VoiceXML 
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browser may be located on the server-side, behind the telephony card, IVR or Gateway 
(PSTN or VoIP). Furthermore, with fat client configurations of multi-modal browsers, 
embedded native VoiceXML browser can appear on the client. Such configuration may 
introduce significant new challenges including, for example, delays, network latencies, 
bandwidth limitations and network overload when loading data files associated to a new 
VoiceXML page; ^d limitation of the capabilities of the local engine (500 words 
vocabulary), etc. These considerations are important and should not be underestimated 
when designing a particular solution. However, such issues are beyond the scope of this 
application. 

It is to be further xmderstood that the term "server-side" or "server-centric" 
processing refers to processing that is executed at the level of a network application 
server (e.g., web apphcation server) and not at the level of an earUer piece of middleware. 
As a simple analogy in the web/HTML world, we would consider Javascript, Applets and 
static long HTML pages (or even DHTML pages) as cHent-side processing and CGI, 
JSP/Beans, ASPs and servlets as server-side processing. 

Therefore, in the present case, cUent-side processing impUes that the reusable 
dialog components are loaded "as is" by a Voice browser/platform to provide the voice 
interaction to the user. Server-side processing impUes that the reusable dialog component 
run at the level of the web application server to contribute to the generation of the next 
VoiceXML page shipped and loaded by the VoiceXML browser. 
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Advantageously, as explained below, reusable VoiceXML component frameworks 
according to the present invention remain within the VoiceXML specifications and 
require no modification of the VoiceXML interpreter (except for features such as 
dynamic compilation of grammars wherein the present invention offers mechanisms for 
5 extending VoiceXML to provide such functionaUty). 

A Reusable VoiceXML Dialog Components 

Fig. 1 is a diagram of a system and method for implementing reusable VoiceXML 
dialog components according to an embodiment of the present invention. In this 
j embodiment, reusable dialog components are built using VoiceXML subdialogs and 

ii 

^ 10 passing arguments through ECMAScript objects. ECMAScript is an object-oriented 
progrmnming language for performing computations and manipulating computational 
objects within a host environment The framework of Fig. 1 is compatible with any 
existing VoiceXML interpreter, independently of specific implementation platforms. 

A <subdialog> element invokes a "called" dialog (referred to as a subdialog) 
15 identified by its src attribute in the "calling" dialog (the src comprises the URI of the 
<subdialog>). A <subdialog> field item is similar to a function call: it invokes another 
dialog on a current page, or invokes another VoiceXML document and returns an 
ECMAScript Object as its result. In accordance with the present invention, subdialogs 
allow the reuse of a common dialog and are used to build libraries of reusable 
20 applications. 



0 
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Referring now to Fig. 1, a VoiceXML browser 10 receives a VoiceXML page 1 1 
from a Web server 12 using, e.g., HTTP. The VoiceXML page comprises one or more 
< subdialog> and < script > elements. A plurality of reusable VoiceXML dialog 
components 13 maybe maintained client-side in local libraries 14. Further, reusable 
VoiceXML dialog components 13 may be maintained and retrieved from a remote 
repository 15. The repository 15 can be a public resource or a server-side (web server) 
repository adapted to the current VoiceXML appUcation by the developer. 

In the framework of Fig. 1, an ECMAScript parameter object 16 is created for 
each reusable VoiceXML dialog component. The parameter objects 16 are populated 
with the appropriate parameter values. For task reusable VoiceXML dialog components, 
all the parameters are optional. For template reusable VoiceXML dialog components, 
additional configuration parameters are needed (e.g. the collection of menu items which 
will be rendered by a navigation menu reusable VoiceXML dialog component). The 
ECMAScript parameter object 16 is characterized by the use of ECMAScript pm-ameter 
objects as containers that provide: default prompts and other object-specific resources, 
constructor that combines default and application-specific parameters, common methods 
for manipulating parameter content. 

The Reusable VoiceXML dialog components 13 are invoked via the 
<subdialog> tag in a VoiceXML page 11. The parameter objects 16 are called from 
an ECMAScript <script> in the VoiceXML page 1 1 . The VoiceXML browser 10 
comprises a ECMAScript host environment. Each reusable VoiceXML dialog 
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component 13 receives its respective ECMAScript parameter object 14 which is created 
as described above. Passing only one parameter in the <subdialog> simplifies the 
readabihty of the code since the amount and complexity of parameters even for a single 
reusable VoiceXML dialog component can be enormous. The execution follows the 
VoiceXML specifications. 

The reusable VoiceXML dialog components 13 are implemented as standard 
VoiceXML documents referenced by the src attribute of the VoiceXML 
<subdialog> . The results of the reusable VoiceXML dialog components 13 are 
retumed in the return variable of the associated ECMAScript parameter objects 16. 
Advantageously, this fi*amework readily fits the Natural Language Semantics Markup 
Language for the W3C speech interface (see, e.g., www.w3.org/TR/nl-specs/), 

As noted above, a Ubrary of VoiceXML reusable dialog component may reside on 
the server repository 15 for dynamic HTTP access at execution, or may reside locally to 
the VoiceXML browser 10. The local library may be downloaded and updated &om the 
same server repository 15. The associated parameter object files are available (same 
name, . j s extension instead of . vxml). 

Task-type reusable VoiceXML dialog components can be directly used without 
customization/configuration. However, the components can also be downloaded in 
advance by the application developer for customization of the associated parameter object 
files and adaptation of the components. When customized, it is important to update the 
src arguments, especially for the parameter object files. 



YOR9-2001-0463US (8728-525) 



-17- 



Template-type of reusable VoiceXML components require configuration of the 
associated parameter object files. Therefore, they should be downloaded, adapted and the 
src arguments should be updated. 

It is to be appreciated that the repositories 14, 15 may further comprise encompass 
default grammars and audio prompts to support the basic behavior of the reusable 
VoiceXML dialog components 13. Further, the system can provide libraries or 
repositories of reusable ECMAScript functions that may be used in the .j s files. Until 
supported by VoiceXML, the repository 15 or other servers can provide dynamic 
grammar compilers and audio prompt editors, which can also be ported to the voice 
browser client platform. 

It is to be further appreciated that the amount of customization/adaptation that is 

needed can be limited by providing rich libraries expanding on the reusable components 

identified in the W3C reusable dialog component requu-ements. In addition, standardized 

sets of such subdialogs ("reusable VoiceXML dialog modules") can be installed on the 

Voice browser side as libraries that are available to the user. 

The following skeleton example illustrates the use and call of reusable VoiceXML 
dialog components. To begin, the following code (myComponentExample . vxml) 
illustrates a VoiceXML skeleton page with intemal calls to a reusable VoiceXML dialog 
component: 

<?xml versions "1.0"?> 

<vxml versions "2. 0"> <!-- File myComponentExample. vxml -~> 

<!-- simpComponent : object for passing parameters to subdialog *-> 
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<scrlpt src="siiapCoinponent. js"/> <!-- include object class definition 

<!-- instance of simpleCoiaponent --> 
<script> 

var mySimpConxponent = new siznpCoinponent(<!--ECM2^Script 
specification of return parameters 

< ! --ECMAScript instantiation and passage of configuration 
parameters --> 

); 

</script> 
<!-- — 

<form ids:"myComponentExaznple"> 
<block> 

<prompt>This is a demo of calling simple reusable 

VoiceXML 

dialog con^onent </prompt> 

</block> 

<!-- sixirple Component Example --> 

<subdialog name= " simpComponentExample " src= " simpComponent . vxml " > 
<param names"MP" exprs"mySimpComponent"/> 

<filled> 

<var names "siyResult" 

exprs " siiopComponentExainple • result" /> 
<prompt>You have selected: <value 
expr s "myResul t " / >< /promp t > 
</filled> 
</subdialog> 
</form> 
</vxml> 



It is to be appreciated that this formulation supports any type of reusable dialog 
components such as task or template type components. 

The following code ( simpCoiriponent . vxml) illustrates a corresponding 

VoiceXML skeleton page comprising the reusable VoiceXML dialog component: 

<?xml version="l,0"?> 

<vxml versions " 2 . 0 " > < ! - - File simpComponent . vxml - - > 

<form ids" Component 

<!-- parameter object --> 
<var names "MP" /> 
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<!-- local vars - - > 

<var names "loc all" expr="MP.iteml"/> 
<l--any other local value asslgxuoent --> 

<var names: "result" expr=""/> <! --Declaration of results 

- -> 

<!-- Reusable component dialog logic code--> 

<!-•" This includes possible use of dynamic audio sources as 

allowed by VoiceXML 2.0. --> 

<!-- It may also require support of dynamic grammar generation 
not yet supported in VoiceXMXi 2.0 (see below for alternative) 
<grammar srcs" javascript sMP. grammar" /> 
</form> 
</vxml> 



For template as well as some task objects, mechanisms are preferably employed to 
support dynamic grammar compilation. One example is illustrated below with an 
example reusable VoiceXMXL dialog component referred to as simpNavMenu . vxml . 



The following code ( simpComponent . j s ) illustrates a corresponding 
mechanism to pass parameters, configuration to and fi-om the reusable VoiceXML dialog 
component: 

// File simpConqponent . js 

// ECMAscript functions to create prompts and other object- specific 
// resources (grammars etc..) . (Optional) 
// For example : 

function makeFullPrompt (p_prompt, p_listl, p_list2) { 
var fullPrompt = p_prompt; 
for (var index in p_listl) { 

fullPrompt = fullPrompt + p listl [index] + «; 

} 

fullPrompt = fullPrompt + 'or 
for (var index in p_list2) { 

fullPrompt = fullPrompt + p list2 [index] + » . » ; 

} 

return fullProi^pt 

} 

function mafceFull6raia(base, p_llstl, p_list2, trailer) { 
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var fullGram = ' ■ ; 

for (var index in p_listl) { 

fullGram = fullGram + p listl [index] + » I 

} 

for (var index in p_list2) { 

fullGram= fullGram + p list2 [index] + • I ■ ; 

} 

// remove trailing separator 
if (fullGram != • ') 

fullGram s base + fullGram. s\ibstring(0, fullGram. lengtli- 3) + trailer; 

return fullGram 

} 

// Mandatory object container tliat combines default and application- 
// specific parameters, create default prompts, grammars and other 
// object specific and resources and provides common methods for 
// manipulating parameter content, 
function simpComponent ( 

p_compTxT 

) { 

// methods 

this.mFPrompt = makeFullPrompt; 
this.mFGram = makeFullGram; 

// constants 

this.constl =0; 

// mandatory properties 

this.compTxt = p_compTxt; 

// optional properties and resources (grammars) 

this.compGram = new Array ( 'textone*, 'Texttwo', 'Textthree' ); 

// internal variables 

this.promptCurr_txt = 'Default text is:'; 

this.helpTxt = this .mFPrompt { 'You can say: ', p_compTxt, 

this.compGram) ; 

this, noma tchTxt = 'I didn\'t understand you. ' + this.compTxt; 
this.noimpTxt = »I didn\ ' t hear you. » + this.compTxt; 

// dynamic grammar source (needed only for ten^late-type of reusable 
VoiceXML dialog components. 

this.grammara=this .mFGr am ( 'grammar GR;\npublic <top> = ', p_compTxt, 
this . coxnpGram, ' ; \n ■ ) ; 
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// optional properties 

thls.promptlinit txt = ■Please do something useful. 

this, prompt lcurr_txt = this .promptCurr_txt; 

thls.promptlcansay_txt s <*; 

this .prompt Unit aud = 'promptlinit.wav*; 
this.promptlcansay_aud = 'promptlcansay.wav"; 

this . prompt 2 lnit_txt = 

this .prompt 2 curr_txt = this .promptCurr_txt; 
this.pros^t2cansay_txt » 

this, prompt 2 lnit_aud = 'prompt2inlt.wav'; 

thls.prompt2cansay_aud = ■prompt2cansay.wav'; 

thls.helpltxt = this.helpTxt; 
thls.help2txt s this.helpTxt; 

this.helplaud = • helplaud.wav ' ; 
this .help2aud = *help2aud.wav' ; 

this .noma tchltxt » this .noma tchTxt; 
this.nomatch2txt - thls.nomatchTxt; 

this . nomatchlaud s ■ noma tchlaud . wav ' ; 
this, noma tch2aud = ■nomatch2aud.wav'; 

thls.nolxnpltxt » this.nolmpTxt; 
thls.nolmp2txt = this .nolii^Txt; 

this • nolmplaud = ■ noln^laud . wav ■ ; 
thls.nolxnp2aud s ■nolmp2aud.wav'; 

thls.filledXxt = 

thls.fllledAud = ■ filledAud.wav ■ ; 

this.errorTxt » 'Sorry, must exit due to an execution error. Call back 
later, please. ■ ; 

thl s . er r or Aud s ■ er r or Aud « wav ■ ; 

} 

The above myComponent Exampl e . vxml illustrates the creation of an 
ECMAScript parameter object (mySimpComponent) for each reusable VoiceXML dialog 
component. The parameter objects are populated with the appropriate parameters. 
Again, as noted above, all parameters are optional for task reusable VoiceXML dialog 



YOR9-2001-0463US (8728-525) 



-22- 



components and for template reusable VoiceXML dialog components, additional 
configuration parmneters are mandatory (e.g. the collection of menu items which will be 
rendered by a navigation menu reusable VoiceXML dialog component). 

The above simpComponent . j s illustrates the use of ECMAScript parameter 
objects as containers that provide: (i) default prompts and other object-specific resources; 
(ii) constructor that combines default and application-specific parameters; and (iii) 
common methods for manipulating parameter content. 

The reusable VoiceXML dialog components are invoked via the <subdialog> 
tag. Each reusable VoiceXML dialog component is passed its respective ECMAScript 
parameter object created as described above. It is to be understood that the above 
example illustrates passing only one parameter in the <subdialog> for purpose of 
simplifying the readability of the code but that the amount and complexity of parameters 
even for a reusable VoiceXML dialog component will vary. Advantageously, the 
execution flow follows the VoiceXML specifications. 

As shown in the exemplary program code above, the reusable VoiceXML dialog 
components are implemented as standard VoiceXML documents referenced by the src 
attribute of the VoiceXML <subdialog>. The simpComponent . vxml illustrates a 
skeleton reusable VoiceXML dialog component (simpComponent). The results of the 
reusable VoiceXML dialog component is retumed in the retum variable of the associated 
ECMAScript parameter object. 
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The following are detailed examples of the implementation of a task type 
(simpDate . vxral) and a template type (sirapNavMenu . vxml) reusable VoiceXML 
dialog components. 

The following VoiceXML file (MyTest .vxml) illustrates a file that calls two 

reusable VoiceXML dialog components: 

<?xml versions "1.0"?> 

<vxml version="2 . 0"> 

simple date: object for passing parameters to subdialog 

<script src="simpDate. js"/> <!-- include object class definition --> 

<!-- instance of simple date 

<script> 

var mySimpDate = new simpDate (}; 
</script> 

<!-- navigable menu: object for passing parameters to subdialog 

<script src="siiapNavMenu. js"/> <I-"- include object class definition 
--> 

<!-- instance of navigable menu 
<script> 

var myNavMenu = new simpNavMenu( 

new Array ( • opera * , 

' concert ■ , 
'exhibition' , 
' theatre ' , 
■musical' ) 

); 

</script> 
<!- — 

<form id="myTest»> 
<block> 

<prompt>Hi! This is a demo of a simple date object . </p romp t> 
</block> 

<!-- simple date --> 

< subdialog name="simpDateExample" srcs" simpDate. vxml "> 
<param name="MP" expr= "my SimpDate "/> 
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<filled> 

<var names "myResult" expr="simpDateExaiaple. result "/> 
<prompt>You have selected: <value expr="inyResult"/></proii^t> 
</filled> 
</subdialog> 

<!„ 
<block> 

<proa^t>Hi! This is a demo of a navigable menu object .</pron^t> 
</block> 

< 1 - - navigable menu - - > 

<subdialog name="navMenuExample" src="simpNavMenu.vxml"> 
<param namesOMP" expr="myNavMenu"/> 

<filled> 

<var names "myResult" exprs"navMenuExample.result*'/> 
<proiapt>You have selected: <value expr="myResult"/></proit^t> 
</filled> 
</subdialog> 

<!-- — 
</f orm> 
</vxml> 



The following code illustrates a task-type reusable VoiceXML dialog component 
(simpDate.vxml): to collect a date input: 
<?xml versions"! •0"?> 
<vxml versions "2 • 0"> 

<form id="simpDateInput"> 
<!-- parameter object --> 
<var name="MP"/> 
<!-- local vars - -> 
<var name="result" expr=:"»»"/> 

<field names "dateInputField"> 
<prompt counts "1"> 

<I-- VXML 2.0 solution: dynamic audio source 
<audio src=s"MP .promptl_aud"><value 
expr= "MP .promptl_txt"/></audio> 
</prompt> 

<prompt counts "2 "> 
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<audio src="MP.proinpt2_aud"><value 
expr="MP •prompt2_txt«/></audio> 
</proiiipt> 

<grain]nar src="simpDate.grain"/> 

<help count="l"><audio src="MP.helplaud"><value 
expr= "MP.helpltxt"/></audio></help> 

<help counts="2"><audio src= "MP. help2aud">< value 
expr= "MP . help2 txt " / ></audio></help> 

<nomatch count="l"><audio src="MP.nomatchlaud"><value 
exprs "MP . noma tchl txt " / ></audio></noiaatch> 

<noinatch count="2"><audio src="MP.nomatch2aud"><value 
expr = "'MP . noinatch2 txt " / ></audio>< /noiaatch> 

<noinput count="l"><audio src="MP.noin^laud"><value 
expr="MP*noimpltxt"/></audio></noinput> 

<noinput count=«2"><audio srca:"MP.nointp2aud"><value 
expr = "MP . noimpl txt " / >< /audio >< /noinpu t > 

<error><audio src=: "MP . errorAud" xvalue 
expr = " MP . er r orTx t " / >< / audio > < exi t / >< / error > 

<filled> 

<pro]npt>< audio src= "MP • filledAud" xvalue 
expr="MP . f illedTxt"/></audio></proittpt> 

<assign name="result" exprs"dateInputField"/> 
<retum nainelist="result"/> 
</filled> 
</field> 
</fonn> 

</vxnil> 

The following program code illustrates a template-type reusable VoiceXML 
dialog component (simpNavMenu • vxml) to present a select menu. As noted above, 
template type reusable VoiceXML dialog component require support for dynamic 
grammar compilation to build on the fly the grammar of the items to select in a dynamic 
menu. 



<?xml versions"!. 0"?> 
<vxml version="2 . 0"> 
<fonn id="navMenu"> 
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<I-- paraxaeter object --> 
<var names: "MP" /> 
<!-- local vars --> 

<var naitie="maxSelIndex" expr=: "MP . selTxt . length - l«/> 
<var names "curSellndex" expr="0"/> 
<var names "result" expr="«'"/> 
<var names "curl temTxt" e3qpr="''"/> 

<!~- generate actual prompt 
<block names "setActProiDpt"> 

<if cond="MP.proiKptlcurr_txtls' •"> 

<assign name="curItemTxt" expr="MP.promptlcurr_txt + 
MP. selTxt [ cur Sel Index] "/> 
</if> 
</block> 

<field names "navS elect "> 
<prompt counts "1"> 

<!-- VXML 2,0 solution: dynamic audio source 
<audio expr="MP .promptlinit_aud"><value 
expr= "MP . prompt linittxt "/ ></audio> 
< value expr="curItemTxt"/> 
<audio exprs "MP . promptlcansay__aud" xvalue 
exprs"MP.promptlcansay_txt"/></audio> 
< /prompt > 



<prompt counts "2 "> 

<audio exprs "MP . promp t2 ini t_aud" xvalue 
exprs " MP . promp 1 2 ini t_tx t " / >< / audio 
<value expr="curItemTxt"/> 
<audio expr="MP.prompt2cansay_aud" xvalue 
eacprs "MP . pron^ t2 cansay_txt "/ X /audio 
</pronipt> 

<!-- proposed VXML extension: dynamic grammar --> 
<grammar srcs " j avascript : MP • grammar" / > 

<help counts "l"><audio exprs "MP .helplaud" xvalue 
exprs "MP . help 1 txt "/></ audio >< /help > 

<help counts'* 2 "xaudio exprs "MP. help2aud" xvalue 
expr="MP,help2txt"/x/audio></help> 

<nomatch counts "i«»><audio expr s "MP. noma tchlaud" xvalue 
expr="MP.nomatchltxt"/></audiox/nomatch> 

<nomatch counts"2"><audio expr= "MP, noma tch2aud" xvalue 
e3q>rs"MP . nomatch2 txt " /x/audiox/nomatch> 

<noinput counts "i"><audio expr= "MP. noimplaud" xvalue 
e3cpr="MP.noimpltxt"/></audiox/noinput> 

<noinput counts"2"><audio exp r s "MP. noimp2aud" xvalue 
exprs "MP . noin^ltxt " /></audio></noinput> 

<error><audio exprs "MP . errorAud" xvalue 
exprs"MP.errorTxt"/x/audioxexit/x/error> 
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<filled> 

<if cond="navSelect==MP.selGram[MP.nextl "> 
<if cond=s"curSelIndex < maxSelIndex"> 

<assign name=:"curSelIndex" exprs"curSelIndex+l»/> 
<else/> 

<assign naiae=" cur Sel Index" expr="0"/> 
</if> 
<else/> 

<i£ conds"navSelecb==MP.sel6ram [MP. previous] "> 
<if cond=" cur Sel Index > 0"> 

<assign naine="curSelIndex" expr="curSelIndex-l"/> 
<else/> 

<assign names "curSellndex" exprs:"iaaxSelIndex"/> 



<if cond="navSelect==MP.selGrani [MP. select] «> 

<a8sigm names ■■result" exprs^^MP.selTxt [curSellndex] 
<else/> 

<assign names "result" exprs"navSelect"/> 
</if> 
</if> 
</if> 

<if cond="result!=" ' "> 

<return namelists" result ■■/> 
<else/> 

<assign names "navSelect" exprs«undef ined"/> 
<ass±gn names "set Ac tProici)t" expr="undef ined"/> 



<prompt><audio expr = "MP. filledAud'^x value 
expr="MP. f illedTxt"/></audio></prompt> 

</filled> 
</field> 
</£orm> 

</vxml> 

The following code (simpDate.js) illustrates a parameter object 
declaration for the simpDate reusable VoiceXML dialog component. 

function simpDate ( 



</if> 
<else/> 



</if> 



) { 



// internal variables 
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this .mainPronptTxt = 'Please select a date.'; 
this.helpTxt = this,mainPromptTxt; 

this, noma tchTxt as 'i didn\'t understand you. • + this.mainProxnptTxt; 
this.noimpTxt = 'I didn\'t hear you. ' + this.niainProntptTxt; 

// optional properties 

this.promptl_txt = 'Please select a date.'; 

thi s . promp t l_aud = ■ FAKE • ; 

this.proit^t2_txt » ■ • ; 

this • pron^ t2_aud = ' PAKE ' ; 

this.helpltxt » this.helpTxt; 
this.help2txt s this.helpTxt; 

this . helplaud a ' FAKE ' ; 
this • help2aud s ■ FAKE ■ ; 

this. noma tchltxt = this .noma tchTxt; 
this. noma tch2txt » this.nomatchTxt; 

this . nomatchlaud = • FAKE • ; 
thi s . nomatch2 aud = ■ FAKE ' ; 

this.noimpltxt » this.noimpTxt; 
this.noimp2txt = this.noimpTxt; 

this • noimplaud = ■ FAKE ■ ; 
this . noimp2 aud = ' FAKE ■ ; 

this.filledTxt = 

this.filledAud = 'FAKE'; 

this.errorTxt = 'Sorry, must exit due to an execution error. Call back 
later, please. ' ; 

this . errorAud = ' FAKE ■ ; 

} 

The following code ( s impNavMenu . j s ) illustrates a parameter object 
declaration for the simpNavMenu reusable VoiceXML dialog component: 



function makeFullPrompt (p_prompt, p_listl/ p_list2) { 

var fullProntpt = p_prompt; 
for (var index in plistl) { 
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fullPrompt = fullPrompt + p listl [index] + • . • ; 

} 

fullProitipt = fullPrompt + 'or »; 
for (var index in p_list2) { 
5 fullPrompt = fullPrompt + p list2 [index] + ■ . • ; 

} 

return fullPrompt 

} 

function ma]ceFullGram(base, p listl, p_list2, trailer) { 

10 var fullGram = » * ; 

for (var index in p_listl) { 

fullGram = fullGram + p_listl [index] + ■ | 

for (var index in p_list2) { 
15 fullGrams fullGram + p_list2 [index] + ' | ■ ; 

// remove trailing separator 
if (fullGram 1= ") 

fullGram = base + fullGram. substring (0, fullGram. length- 3) + trailer; 



%«l 20 return fullGram 

III 



} 



function siinpNavMenu( 
p_selTxt 

) { 

25 // methods 

this.mFPrompt = malceFullPrompt; 
this.mFGram = makeFullGram; 

// constants 

this .next = 0; 

30 this. previous =1; 

this .select = 2; 

// mandatory properties 

this.selTxt = p_selTxt; 

// optional property 
35 // VERY SIMPLE: only three navigation phrases are possible 
// they must be specified in this fixed order 
this.selGram = new Array ( 'next^, 'previous S 'select" ); 

// internal variables 

this«promptCurr_txt s 'Your current selection iss'; 
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this.helpTxt = this.mFPronrpt ( 'You can say: p selTxt, this.selGram) ; 
this. noma tchTxt = 'I didn\ ' t understand you, ' + this .helpTxt j 
this •noimpTxt = "I didn\ « t hear you. • + this.helpTxt; 

// dynaislic grammar source 

thi s. gr ammar a this.mFGr am ('grammar 6R;\npublic <top> = p_selTxt, 
this . selGram, ' ; \n « ) ; 



// optional properties 

this .promptlinit_txt 
this, prompt lcurr_txt = this .promptCurr_txt; 
this.promptlcansay_txt = 



this .promptlinit_aud = 
this.promptlcansay_aud = 



this • prompt 2 init_txt = 
this.prompt2curr_txt = this«promptCurr__txt; 
this .prompt2cansay txt = 



this .prompt2init_aud = 
this .pronipt2cansay_aud = 



Please select a cultural event. 



promptlinit.wav' ; 
promptlcansay.wav' ; 



prompt2init.wav' j 
p r omp 1 2 c ans ay . wav ' ; 



this.helpltxt » this.helpTxt; 
this.help2txt = this.helpTxt; 

this.helplaud = • helplaud.wav » ; 
this.help2aud » ■help2aud.wav'; 

this.nomatchltxt = this, noma tchTxt; 
this*nomatch2txt = this, noma tchTxt; 

this.nomatchlaud = 'nomatchlaud.wav'; 
this .noma tch2aud ^ ■ nomatch2aud.wav ■ ; 

this .noimpltxt = this .noimpTxt; 
this.noimp2txt ^ this .noimpTxt; 

this .noimplaud = 'noimplaud.waV ; 
this.noimp2aud = 'noimp2aud.wav*; 

this . f illedTxt = ' • ; 

this.filledAud = ■filledAud.wav'; 

this.errorTxt - * Sorry, must exit due to an execution error. Call back 
later, please. ■ ; 

this • errorAud = ' errorAud .wav ' ; 
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An exemplary grammar file (simpDate.gram) shovm in Appendix A, specifies the 
associated Dategrammar file. 

Preferably, to implement reusable VoiceXML dialog components within the 
VoiceXML specifications, a VoiceXML platform should support (i) dynamic grammar 
generation and (ii) passing audio as a variable. Currently, VoiceXML does not support 
dynamic grammar generation. In one preferred embodiment, a grammar is dynamically 
compiled outside of the VoiceXML interpreter. For example, <grairanar 
src=" javascript : MP. grammar "/> in simpComponent . vxml can be 
replaced by: <grainmar src=" http : //www* graitimar_generator .example 
/dynamiccompiler?MP.graiiraiarfURI''> or <graramar src= 
"http : //localhost/dynamiccompiler?MP .grammarf URI">, depending 
on whether the compiler is on a server or local. 

The MP . grammar fURI is a javascript fimction that converts the grammar 
parameters in an URI compliant form. Other plug-in mechanisms may be employed. 

Iq accordance v^ith the present invention, it is preferred that a VoiceXML 
specification be added for dynamic grammar generation, hideed, dynamic grammar 
generation is needed for template type reusable VoiceXML dialog components as 
illustrated by simpNavMenu . vxml . Moreover, there are task reusable VoiceXML 



YOR9-2001-0463US (8728-525) 



-32- 



dialog components that may be difficult to implement without dynamic-grammar support, 
e.g. spoken-and-spelled name. 

VoiceXML 2.0 supports the ability to pass audio as variable. This feature is 
useful for implementing audio prompts. Currently, with VoiceXML 1.0, the prompts 
must be generated outside the VoiceXML interpreter and passed by URI similarly to the 
dynamic grammar case. 

There are numerous advantages associated with the framework discussed above 
using reusable VoiceXML dialog components. One advantage is that the reusable 
VoiceXML dialog components are browser independent and do not require any change to 
existing browsers (except for the new VoiceXML requirements proposed herein). 
Further, the task of authoring complex reusable dialog components and modules is left to 
the skilled VoiceXML programmers (e.g., VoiceXML reusable dialog components, 
associated ECMAScript parameter object files and associated default grammars and 
prompts). Moreover, developers who use the reusable VoiceXML dialog components can 
use such components as libraries or as sample code / templates to build more complex 
applications and reusable VoiceXML dialog modules or customize them. 

In other words, reusable VoiceXML dialog components will allow users to leam 
from and re-use standard components as they get deployed. This will have the same 
impact on easing learning and adoption as was seen with HTML where users could see 
how a certain page was written when authoring their sites. Similarly, the use of reusable 
VoiceXML dialog components will be instrumental in the growth of voice appHcations. 
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Another advantage is that the reusable VoiceXML dialog component framework 
described above meets the W3C VoiceXML and reusable dialog component 
specifications and recommendations. Further, intemationaUzation and localization can 
be supported by providing different versions of the reusable VoiceXML dialog 
components and adapting the parameter object files accordingly. This can be provided in 
the above mentioned libraries or repositories or the components can easily be adapted by 
a developer starting from the sample reusable VoiceXML dialog components. These are 
examples of reusable VoiceXML dialog modules as noted above. 

The simultaneous activation of the components is currently a feature beyond the 
scope of VoiceXML. In accordance with the present invention, however, mechanisms are 
provided to support this feature (e.g., mixed initiative and fi"ee flow dialog) as discussed 
below. 

Furthermore, in a reusable VoiceXML dialog component firamework as described 
above, retum values support the NL formats, error/exception handling are expUcitly 
addressed (as illustrated in the skeleton and examples above), prompts are set as part of 
the associated parameter object file and the associated error and exception handling logic 
is explicitly implemented in the reusable VoiceXML dialog components. The present 
invention supports coordination between language and component features since the 
reusable VoiceXML dialog components are declarative and the behavior can be 
immediately determined by direct examination. Further, staying within the VoiceXML 
framework guarantees a well predictable scoping of global commands. In addition, to 
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provide a consistent user experience, it is preferred that a reusable dialog component 
framework according to the present invention follows scopes as for subdialogs. 
Component composition is also supported as described earher and further because of the 
declarative nature of the reusable VoiceXML dialog components and modules. 

Further, with respect to the W3C updated requirement proposal, reusable 
VoiceXML dialog components according to the present invention preferably support : (i) 
returning the control to the calling routine after execution of the component (by design of 
subdialogs); (ii) multiple invocation; (iii) multiple appUcation can point to the same 
subdialog; (iv) reusable component can be configured at execution through an associated 
ECMAScript parameter object and no compilation is required except for grammar 
compilation as discussed earlier; (v) the interface is clearly standardized through the 
result object and the ECMAScript parameter object; and (vi) platform dependency is 
trivially determined by inspection of a reusable VoiceXML dialog component: if the 
reusable component does not contain an <ob j ect > tag, it is platform-independent but if 
the reusable component does contain an <ob j ect > tag, then another framework as 
described herein may be implemented. 

The requirement for simultaneous / parallel activation of the components is more 
complex and is currently a feature that is currently beyond the scope of VoiceXML and 
currently addressed herein through shared grammars across form items. This feature is 
discussed in detail below (e.g., mixed initiative and fi'ee flow dialog) wherein 
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mechanisms are added to the existing W3C requirements to support mixed initiatives: 
context sharing across reusable dialog components (subdialogs as well as objects). 

When developing client-side VoiceXML applications, it may be difficult to build 
a voice-based application without access to dynamic data sources like dynamic grammar 
compilation and dynamic access to databases (via HTTP server or by ODBC, SQL, 
etc. , .). In accordance with the present invention, components and mechanisms are 
provided to access dynamic data sources. 

B Reusable VoiceXML Dialog Beans 

In addition to building client-side reusable VoiceXML dialog components and 
modules that satisfy the existing VoiceXML framework using existing VoiceXML 
browsers and web infrastructures, in accordance with another embodiment of the present 
invention, server-side solutions with dynamic generation of the VoiceXML pages are 
preferably employed to support dynamic manipulation of prompts, dynamic grammar 
compilation and dynamic access to data sources . 

A common issue with server side reusable dialog components is the selection of a 
programming language - environment (Java, Perl, Python, PHP, C, VisualBasic, ...). In a 
preferred embodiment, the dialog component library is programming language agnostic in 
order to achieve wide acceptance and reusability. There are several interlingual 
frameworks that allow integration of software components that are implemented in 
different programming languages. Some of them are limited to MS Windows (COM, 
.NET runtime), some are limited to Linux Gnome (Bonobo), but other are platform 
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independent (CORBA Components, UNO, XPCOM). Because inter-process 
communication is not necessary for the reusable dialog components, UNO and XPCOM 
seem to be more suitable for this task. 

Fig. 2 is a diagram of a system and method for implementing reusable VoiceXML 
components according to another embodiment of the present invention using a 
server-centric framework. Generally, the server-centric framework in Fig. 2 comprises a 
Web appUcation server 20 comprising a page generation module 21 . The page generation 
module 21 dynamically generates a VoiceXML page 22 that is processed by a VoiceXML 
browser 23. To build a VoiceXML page, the page generation module 21 accesses (i) 



Cil 1 0 reusable VoiceXML server-side components from repository 24 and^r (ii) dynamic data 

! from database 25. 

f-|i 

f 4 In general, a method for server-side generation of a VoiceXML page according to 



one aspect of the invention comprises the following steps. Initially, an intermediate 

VoiceXML code is constructed using predefined reusable dialog components 24. Then, 
15 data from the dynamic data sources 25 is inserted into the intermediate code. Finally, the 

resulting pure VoiceXML code (with or without <ob j ect > or < subdialog> 

elements) is sent to the browser 23. 

It is to be appreciated that various methods may be used for implementing 

server-side reusable VoiceXML dialog components. For instance, server-side reusable 
20 VoiceXML dialog components may comprise VoiceXML objects that are created by 

reusing the client-side reusable VoiceXML dialog components and modules as described 
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above in Section EA. Further, in another VoiceXML framework, the VoiceXML objects 
may comprise beans, whereby a markup template processing engine (e.g., JSP(JavaServer 
Pages), ASP(Active Server Page), PHP(Personal Home Page Tools)) provides a 
framework to dynamically generate the VoiceXML pages using the beans. In addition, as 
explained below, service beans can be used to dynamically compile grammars and 
dynamically manipulate prompts. 

In a preferred embodiment of the present invention, the server-centric framework 
illustrated in Fig. 2 comprises a JSP reusable VoiceXML beans framework, where the 
beans are similar in function to the cUent-side reusable VoiceXML dialog components (or 
modules) as discussed above. The details of a preferred JSP/beans framework are 
described in, e.g., U.S. Patent AppUcation Serial No. 09/837,024, filed on April 18, 
2001, entitled "Systems and Methods For Providing Conversational Computing Via 
JavaServer Pages and Javabeans", which is commonly assigned and incorporated herein 
by reference. Generally, JSP is method for controUmg the content or appearance of Web 
pages through the use of servlets, which are small programs that are specified in the Web 
page and run on the Web server to modify the Web page before it is served to a 
requesting chent. U.S. Serial No. 09/837,024 describes a server-centric solution based on 
Java Server Pages (JSP) with embedded reusable VoiceXML dialog beans (JavaBeans), 
wherein VoiceXML pages are dynamically generated by JSPs that call beans. Beans are 
classified as (1) VoiceXML dialog beans that generate VoiceXML subdialogs (a simple 
framework can generate all the reusable dialog components enumerated in the W3C 
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reusable dialog component requirements) and (2) service beans that perform other system 
tasks (such as dynamic access to backend (via HTTP server or by ODBC, SQL, etc.), 
determination of client properties, compilation of dynamic grammars etc). 

The following skeleton example describes the use and call of reusable VoiceXML 
dialog beans. In particular, the following JSP skeleton page ( con^onent_vxmi . j sp ) is 
responsible for creating and rendering reusable VoiceXML dialog beans: 



<%®page sessions" true" languages" java" contentTypes"text/vxnil" %><?xml 
^'^^ versions " 1 . 0 " ? > 

M <!-- The application business logic makes sure that the Form container 

IJl 10 contains the DSlot speech object, such that calling renderFaceO on the 
III Form renders the VoiceXML content o£ the DSlot date bean.--> 



<jsp:useBean id=" component" scopes "session" 
classs " exaxE^le • vxmlserver • ui • Form" / > 

<vxml versions "2. 0"> 
15 <% coxi^onent.renderFace(out) ; %> 
</vxml> 



In JSP/beans framework, reusable VoiceXML dialog beans are embedded in JSP 
pages that are responsible for bean creation and rendering. Since a full-fledged 
object-oriented procedural language (Java) is used for implementing the server-side 

20 reusable VoiceXML dialog beans, the speech objects are readily reusable through 
inheritance or aggregation. Preferably, the rendering follows a MVC 
(Model- View-Controller) paradigm, where the UI Tenderers, acting as Views, encompass 
markup-specific rendering code (renderFace) and the Model maintains the 
modahty-independent state of the object. Each bean comprises a "rendering face" for 

25 each modality supported by the server, which allows each bean to retum its results (in 
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response to a JSP request) in the appropriate modality. A Database stores the object 
instances and data associated with the various beans. 

The "rendering face" of a given bean comprises a method that inspects (by 
introspection) the type of channel used for access and then, upon determining the channel 
5 type, e.g., VoiceXML access, the rendering face method returns a VoiceXML portion of 
the user interface to embed in the calling JSP (and in the VoiceXML pages) that the JSP 
dynamically generates. 

II Fig. 3 illustrates a system and method for implementing VoiceXML beans 

according to the present invention. In Fig. 3, a web appUcation server 30 generates 
10 modality-specific pages 3 1 (e.g., VoiceXML pages). The Web application server 30 
comprises a database 32 of reusable service beans and reusable dialog beans having 
C'l rendering faces for VoiceXML. The VoiceXML pages 3 1 are dynamically generated by 

compiling a CML (conversational markup language) JSP page 33 usmg a CML skeleton 
framework and the beans 32. Database 34 stores CML JSP pages 33 and backend content 
15 and business logic for the applications. Database 35 stores the object instances and data 
associated with the various beans 32. 

More specifically, in the framework of Fig. 3, authoring begins with drafting a 
CML (conversational markup language) skeleton associated with a CML-JSP page 33. 
Preferably, the CML skeleton comprises program code that represents a 
20 document/application in a single authoring, modahty-independent manner. The CML 
skeleton is generated and stored, e.g., in the database 34. In a preferred embodiment. 
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CML comprises m interaction-based, modality-independent framework for building 
multi-modal applications. One embodiment of an interaction-based programming model 
that may be implemented herein is described, for example, in U.S. Patent application 
Serial No. 09/544,823, filed on April 6, 2000, entitled: ''Methods and Systems For 
Multi-Modal Browsing and Implementation of A Conversational Markup Language^\ 
which is commonly assigned and folly incorporated herein by reference. In general, U.S. 
Serial No, 09/544,823 describes a new programming paradigm for an interaction-based 
CML (conversational markup language) in which the application content (business logic 
and backend access) is separate from user interaction (CML is now referred to as iML 
(interaction markup language). More specifically, the CML programming model 
sq>arates application programming into content aspects, presentation aspects and 
interaction aspects. IML preferably comprises a high-level XML-based language for 
representing "dialogs" or "conversations'' between user and machine, which is preferably 
implemented in a modality-independent, single authoring fonnat using a plurality of 
"conversational gestures." Conversational gestures comprise elementary dialog 
components (interaction- based elements) that characterize the dialog interaction with the 
user and are bound to the data model manipulated by the user. Each conversational 
gesture provides an abstract representation of a dialog independent from the 
characteristics and UI offered by the device or application that is responsible for 
rendering the presentation material Li other words, the conversational gestures are 
modality- independent building blocks that can be combined to represent any type of 
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intent-based user interaction. A gesture-based IML, for example, allows an application to 
be written in a manner which is independent of the content/application logic and 
presentation (i.e., gesture- based IML encapsulates man-machine interaction in a 
modality-independent manner). 

The exemplary CML JSP page 33 comprises a plurality of components (program 
code modules) such as an XFORMS portion 33a (generally, a data model declaration), 
various sequences of gestures 33b, 33d (CML interaction pages that call gesture beans) 
and a sequence of service calls 33c (which call service beans). In a hybrid case, the CML 
JSP page 33 may also comprise some object calls. It is to be understood that the 
sequence and use of such components will vary for a given CML JSP page based on the 
application. The XFORMS component 33a of the CML JSP page 33 specifies one or 
more data models for user interaction. The XFORMS component 33a of the CML JSP 
page 33 declares a data model for the fields to be populated by the user interaction that is 
specified by the one or more gestures 33b, 33d. In other words, the CML interaction page 
33b, 33d can specify the portions of the user interaction that is binded on the data model 
portion 33, XForms is compatible and supports XSchema, which may be implemented 
herein. It is to be understood that other languages that capture data models and 
interaction may be implemented herein. 

Those portions of the CML skeleton comprising the sequences of gestures 33b, 
33d comprise code (scripts, tags, etc.) that access/call corresponding gesture beans in the 
database 32. In one embodiment, the database 32 comprises a finite set of gesture beans. 
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wherein the set of gesture beans comprises at least one gesture bean for each of the 
fundamental CML conversational gestures described in the above-incorporated U.S. 
Serial No. 09/544,823, and possibly a finite set of reusable dialog components that are 
built from the elementary finite set of gesture beans. 

Furthermore, the gesture beans 32 may comprise one or more inherited gesture 
beans that are generated firom corresponding gesture beans. An inherited gesture bean 
may be used to provide cosmetization or specialized rendering in a target ML associated 
with the corresponding gesture bean. Cosmetization or specialization is a method for 
optimizing an application for a given channel (device , modality or browser) or a class of 
channel (e.g., Nokia cell phones, etc.). For example, specialization may includes 
providing a background for a page, changing the layering of a page into frames, 
fragmenting a WML document across multiple deck of cards, specifying the voice 
characteristics for a TTS prompt or an audio prompt to play back, changing the message 
to present to the user when spoken versus the displayed message, skipping a gesture not 
needed in a given modality, etc. This concept is analogous to cosmetized XSL rules for 
the conversational gestures as described in the above-incorporated U.S. Serial No. 
09/544,823. 

Next, those portions of the CML JSP page 33 that correspond to sequences of 
service calls 33c comprise program code that are specified as service bean calls. 
Preferably, the programming model for the service beans is imperative (i.e., the 
conventional model), although any suitable constrained model maybe implemented. In 
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addition, unlike the set of gesture beans which is preferably limited, the number of 
service beans that may be employed is unlimited. The service beans are employed to, 
e.g., provide access to dynamic data, provide access to the backend legacy content, 
provide a mechanism to maintain dialog states, etc. Server beans comprise a rendering 
component which, as explained below, allows the service call to be rendered in CML 
(and possibly a XFORMS) format. In other words, the service beans preferably produce 
CML and/or XFORMS page portions (including possible calls to gesture beans) that are 
inserted in the resulting CML-JSP page prior to rendering of such page. 

There are various advantages associated with a framework based on reusable 
VoiceXML dialog beans as shown in Fig. 3. For instance, the reusable VoiceXML dialog 
beans are browser independent and do not require any change to existing browsers. 
Further, the authoring of complex reusable dialog components and modules is left to 
skilled VoiceXML programmers (e.g., authoring of reusable VoiceXML dialog 
beans (VoiceXML renderers, data models and backend implementation) and grammar 
files. Another advantage is that developers who use the reusable VoiceXML dialog bean 
just need to author VoiceXML templates (JSPs, ASPs, PHPs) that call the rendering code 
of the reusable VoiceXML dialog beans as they can involve other service beans. The 
beans framework exploits the servlet paradigm for web servers / web application servers. 
As such it directly fit into numbers of robust and scaleable web / e-business 
infrastructures. 
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Further, the use of the MVC principle enables extension of the apphcation 
authoring to multi-channel multi-modal applications. In addition, a beans framework 
and server side application can maintain context and enable context sharing and dialog 
flow control. Therefore, this framework is ready to support mixed initiative applications 
when available. 

Another advantage is that reusable VoiceXML dialog beans will allow users to 
learn from and re-use standard components as they get deployed, which will lead to the 
growth of voice applications. 

Further, no extensions are required for voice browser or VoiceXML with the 
server-side VoiceXML generation approach and the reusable VoiceXML dialog beans 
discussed above. A reusable VoiceXML dialog bean framework satisfies the 
requirements and recommendations of the W3C speech framework. 

IIL Dialog Object Interfaces and Behaviors 

This section addresses the use of the <ob j ect > tag within Voice XML for 
encapsulating dialog elements and other service calls whose behavior is implemented via 
platform-specific objects, and enumerates some requirements to ensure that the 
component framework used for dialog objects enables rich context sharing amongst 
various dialog components comprising a conversational application. It is to be 
understood that this discussion goes beyond the current VoiceXML 2.0 specifications 
and execution model. 
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In accordance with an embodiment of the present invention, a framework is 
provided for context sharing objects and subdialogs to support their parallel activation 
and mixed initiative across objects, subdialogs and documents. This functionality is 
preferably achieved by using "re-entrant" objects and or subdialog, meaning that objects 
and subdialogs can be initiated, interrupted, inspected and resumed with a partially filled 
result/state object. These frameworks require extensions of the VoiceXML specification, 
the form interpretation algorithm and the VoiceXML execution model. These frameworks 
provide support for advanced mixed initiative beyond current VoiceXML specification 
that allow mixed initiative for fields of a same form or within a same document. 

hi the VoiceXML framework, the <obj ect > element is used to expose 
platform-specific functionality for use by a VoiceXML application. A <param> 
element can be used to pass parameters to the <ob j ect > when it is invoked. When an 
<obj ect > is executed, it returns an ECMAScript result object as the value of its form 
item variable. 

The use of reusable VoiceXML dialog components as described in the previous 
section support standard VoiceXML implementations. In view of the reusable 
VoiceXML dialog component frameworks presented above, it is preferred that the use of 
binary objects (i.e., non-declarative VoiceXML) be limited as much as possible to the 
implementation of fimctions or behaviors not supported by VoiceXML so as to maintain 
browser interoperabiUty. For example, <obj ect > can be used to implement speaker 
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recognition functions as enrollment, verification and identifications, and other functions 
not currently supported by VoiceXML. 

Similarly, it is preferred that dialog modules which only use functions supported 
by VoiceXML rely on the reusable VoiceXML dialog component and module firamework 
5 described above or the server-side beans framework to remain within the VoiceXML 
specifications. 

A. Object Types 

r;i There are various type of objects that are classified as "service objects" and 

"interaction objects". Service objects comprise objects that do not process the dialog 



Wl 1 0 with the user to affect the dialog, but can rather affect the state of the dialog, the state of 
r . the apphcation or the state of the platform (client or server) or they can process the dialog 

hi 

pi with no direct impact on this dialog other than confirmation of success, failure or 

r!| completion. Examples of service objects include, e.g., objects that check the caller ID, 

check the time of the call, pre-populate a dialog form based on information known about 
15 the user or record and log a whole conversation (as compared to the <record>) 

function. 

Service objects also provide dynamic access to data sources (via HTTP server or 
by ODBC, SQL, etc . . .). The required fimctionality of dynamic data source access objects 
makes it virtually impossible to develop such access objects using VoiceXML. 
20 Therefore, consideration should be given to using some native programming language - 
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preferably the same as the language that the VoiceXML browser is written in. It is also 
possible to imagine these objects as similar to browser plug-ins. 

The current VoiceXML <ob j ect > specifications seem adequate to support such 
services objects. Service objects are similar to the service beans as described above with 
reference to Fig. 3 in a reusable VoiceXML dialog beans framework. 

Further, interaction objects comprise objects that directly process dialogs with the 
user. Interaction objects can be further classified mto "dialog module" objects and "I/O 
processing" objects. Dialog module objects comprise objects that provide partial dialog 
and dialog logic to populate a set of fields. They can provide functions analogous to the 
reusable VoiceXML dialog components or modules as defined above, except that their 
intemal execution model does not have to be limited to the VoiceXML execution model 
and capabilities. For example, a speech biometric verification dialog module can provide 
verification of a user simultaneously based on acoustic verification (speaker verification) 
and content verification (acceptance of the answers to questions). 

It is to be noted that the reusable VoiceXML dialog components (as well as the 
beans) discussed above is analogous to the dialog module object type, using the 
<subdialog> tag mstead of <ob j ect > . Therefore, in accordance with one 
embodiment of the present invention, the expected behavior of both elements is closely 
linked. 

I/O processing objects comprise objects that process user inputs and output to the 
user and populate a set of fields without carrying any dialog or partial dialog with the user 
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or process <param> to produce and output to the user. Examples of I/O processing 
objects include, e.g., a speaker identification I/O processing object that performs 
text-dependent or text-independent speaker identification or an NLG (Natural Language 
Generation) processing object that generates the prompt to feed to a TTS engine on the 
basis of a set of attribute value pairs through <param> . 

Literaction objects bring more demanding requirements. Because of the similarity 
between interaction objects and the subdialogs as described above in the reusable 
VoiceXML dialog component framework, we will discuss issues with execution flow, the 
form interpretation algorithm, etc. for both <subdialog> and <obj ect > tags. 

Preferably, to support mixed initiative according to the present invention, 
interaction objects and subdialogs preferably support parallel activation (e.g., objects that 
can process the next I/O event). Consequently, in accordance with the present invention, 
mechanisms are preferably implemented to (i) decide what object or objects process a 
given event (ii) share events and state (context) between objects and to (iii) switch 
between running objects and pass from one dialog to another (e.g., Stop or interrupt an 
object and save its state, access to state, launch an object in a prescribed state, etc.). This 
goes beyond the level of parallel activation described in the updated W3C reusable dialog 
component requirements, where parallel activation would not play beyond the first 
prompt or entry point in the object. 

Further, to support mixed initiative, interaction objects and subdialogs preferably 
support the use of expert systems to improve processing of particular dialog situations 
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and therefore base a portion of an interaction object on the result of another object that 
was running in parallel. 

hi addition, some level of support is preferred for multi-channel and multi-modal 
rendering. For example, with the JSP/bean approach, the use of the MVC principle and a 
renderFace function enables to reuse the same beans across multiple channel 
(multi-channel applications) with possible synchronization of these channels 
(multi-modal applications): renderFace (out, getModality () , cp) ; where 
getModality ( ) returns the access channel(s) and cp describes the client profile (this 
can be based on the W3C CC/PP working group deliverables). 

Fig. 4 is a diagram of a cUent-centric framework implementing service objects for 
dynamic access to data sources as well as interaction objects and subdialogs. A 
VoiceXML browser 40 processes a VoiceXML page 41 (received from VoiceXML web 
server 42). The VoiceXML script comprises a plurahty of <ob j ec t > and 
<subdialog> elements. The <obj act > elements call service objects (as well as 
interaction objects) for dynamic data access via, e.g., HTTP to an HTTP server 43 or SQL 
(structured query language) to a database server 44. The <subdialog> components 
call interaction objects (reusable VoiceXML components). 

B. Execution and Navigation Flow Issues 

To illustrate the impact of the above requirements, consider the 
VoiceXML mixed initiative specifications - to make a form mixed initiative, 
where both the computer and the human direct the conversation, the form must 
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have one or more <initial> form items (which controls the initial interaction 
in a mixed initiative form) and one or more form-level grammar (which allows 
the fields of a form to be filled in any order and allows more than one field to be 
filled as a result of a single utterance). 

This works well for <f ield> as form items. It does not work for binary 
<obj ect> (e.g., interaction objects) or <subdialog> as form items, when 
mixed initiative is expected across objects (instead of intra-objects). Indeed, as 
specified in the VoiceXML forum, once entered, an object must complete before 
returning the hand to the VoiceXML interpreter. 

Fig. 5 illustrates the execution flow when invoking a sub-dialog 
composed in multiple documents as provided in the VoiceXML 2.0 
specifications. Fig. 5 shows the execution flow of appKcation where a sequence 
of documents D transitions to a subdialog SD and then back. The main dialog 
execution context 50 comprises documents D1-D4 and the subdialog execution 
context comprises documents SDl and SD2. A subdialog invocation occurs in 
document D2, wherein the subdialog executes via documents SDl and SD2, and 
retums after execution. In this execution model, the calling dialog is suspended 
(the execution context in dialog D2 is suspended) when it invokes the subdialog 
SDl , awaiting retum of information firom the subdialog. As such, the execution 
flow in Fig. 5 does not support mixed initiative that would fi-ee the navigation 
flow between SDl, SD2 and D2. For example, there is no known VoiceXML 
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specification for allowing the user to input form items relative to D2 while 
interacting with SDl . 

Indeed, the existing VoiceXML specifications only indicate that if a form 
has form-level granamars, its fields can be filled in any order and that more than 
one field can be filled as a result of a single user utterance. Further, the form's 
grammars can be active when the user is in other dialogs. If a document has two 
forms on it and both forms have grammars that are active for that document, a 
user could respond to a request from one form with information about the other, 
and thus direct the computer to talk about the second form instead. The user can 
speak to any active grammar, and have fields set and actions taken in response. 

This mechanism enables mixed initiative only within a document and rehes on the 
"scope" of active grammars to decide the form item activated by a given user input. 
However, this mechanism does not provide for mixed initiative across documents, nor 
does it provide a mechanism for exiting form items that have their own execution flow 
before completion of the dialog flow. 

We have identified various issues with the current VoiceXML specifications: 

(1) <object>: when an object is running and carrying a dialog with the user 
(whether the object is a dialog module object or I/O processing object), only errors can 
interrupt the execution flow; 

(2) <subdialogs>: When a subdialog is interpreted, the only active grammars are 
those in dialog-scope of the subdialog and the default grammars defined by the interpreter 
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context (e.g. help, cancel). The set of active grammars remains Hmited for all subsequent 
dialogs imtil a <retum> is executed. As explained below, to support mixed initiative in 
accordance with the present invention, a mechanism is preferably provided for sharing 
and aggregating the parent form grammar with its subdialog. 

Further issues of the current VoiceXML specifications are that the <record> 
and <transf er> behavior is uncertain when mixed initiative is to be supported such 
functions can not be escaped by the user (for example during bridging, before returning to 
the interpreter), or combined with another query. Further, with respect to <block> 
behavior, it is unclear whether or not there can be mixed initiative between block form 
items and the other form items of the parent form. 

The execution model for objects is not specified in the current version of 
VoiceXML 2.0 as they are essentially platform specific. Because objects are a priory 
binaries, not available to inspection by the VoiceXML interpreter, it is not possible to 
simply extend the grammar scope and execution flows as we could do with subdialogs. 

C. Re-entrant Objects 

Therefore, in accordance with the present invention, to support mixed initiative, 
objects and subdialogs (e.g., the next generation of reusable VoiceXML dialog 
components or modules) are preferably "re-entrant" or support '*re-entrant" behavior and 
interfaces. 

In accordance with one aspect of the present invention, a re-entrant object or 
subdialog is one that supports launch with a partially filled state. More specifically, an 
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ECMAScript result object can be manipulated prior to launching the object and partially 
filled. Further, the object initiated in its partially filled state (i.e. result object), has an 
execution flow that allows it to continue to collect the missing members of its result 
object. A resuh object according to the present invention differs fi-om the current 
specification since, e.g., the result object may comprise a description of the fiiU relevant 
intemal state of the object (rather than just the final retum values). 

Further, a re-entrant object or subdialog is one that supports an 
execution_f lag that prescribes if the object should: 

(i) retum its object result only when it is completed (this is the current execution 
model of subdialogs and probably the common understanding of the execution model of 
objects, and this could be the default settings for the execut ion_f lag); 

(ii) retum its object result even when it is partially filled, but a particular event has 
been triggered (low confidence, error, etc.); 

(iii) retum its object result or launch an event (this is more an implementation 
choice) when the intemal result object changes; and/or 

(iv) retum its object result or launch a blocking event (this is more an 
implementation choice) when the intemal result object changes (in the latter case, when 
blocked, it should be possible to get and set the members of the result object and resume 
the object execution). 

Further, when the event approach is followed, the re entrant object or subdialog 
preferably can launch blocking events, be blocked, support set and get on the members of 
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its result object and allow its execution to be resumed with a modified partially filled 
result object and continue to collect the missing members of its result object. 

In addition a re-entrant object or re-entrant subdialog is one that allows possible 
VO sharing through an additional object, e.g., allowing an utterance be passed to other 
engines associated to other objects for separate processing when needed. This is 
important if parallel objects use different engines or different data files and properties. 
This can be done as member of the result object. 

Re-entrant objects should now throw not only error . unsupported . ob j ect 
if the particular platform-specific object is not supported, but throw an analogous 
error . unsupported . obj ect . execut ion_f lag, if the selected execution flag 
is not supported. Vendors may decide to revert to a default behavior (probably the default 
execut ion__f lag settings) or wait for the interpreter to handle. 

It is to be appreciated that the <subdialog> specifications do not have to be 
fimdamentally modified. To support re-entrance, the present invention adds an 
execution_f lag argument and modifies the execution flow when the 
execut ion_f lag is not set at its default value. Because of the declarative nature of 
subdialogs, the manipulation of the result object and capabiUty to handle modified or 
partially filled result objects can be readily supported with an upgraded VoiceXML 
interpreter. 

There are various advantages to using re-entrant objects and subdialogs. For 
example, mixed initiative across and between subdialogs, objects and documents 
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becomes possible when supported by the VoiceXML interpreter. Another advantage is 
that parallel and simultaneously objects and subdialogs can be appropriately supported. 

A further advantage is that a user input can be processed through parallel engines. 
This enables, for example, a speech biometric implementation as described earlier where 
acoustic and content based authentication are simultaneously performed. Re-entrance 
also enables to decide on the fly which of a set of parallel engines should process a given 
input. 

It is further possible to involve other expert resources to handle dialog errors in an 
object or subdialog or condition internal processing of an input on the result of other 
parallel objects or subdialogs. In addition, context-sharing between modalities, including 
explicit time-sharing of input events in different modahties becomes possible for 
multi-modal appUcations. 

Experimental Results 

To validate the proposed framework, we have developed a set of re-entrant 
reusable dialog components using the IBM form-based dialog manager (FDM) as 
described in the references by Papineni, et al. "Free-flow dialog management using 
forms", Proc. Eurospeech, Budapest, 1999, and Davies, et al., "The IBM conversational 
telephony system for financial appHcations", Proc. Eurospeech, Budapest, 1999. The set 
of components essentially covers the set of reusable dialog requirements for Voice 
Markup language (www.w3.org/TR/reusable-dialog-reqs) and a series of transaction-level 
reusable dialog components. In this setting, the VoiceXML browser simultaneously loads 
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components (<object>) that are loaded with a same activation scope as argument. This 
goes beyond the current VoiceXML specifications. As VoiceXML does not really specify 
context sharing, this implementation limits context sharing and mixed initiative between 
simultaneously activated objects; not between objects and the VoiceXML appHcation. 
5 This is a simplification of the appUcation, clearly not a limitation of the firamework. 

Actually, we separately extended support of context sharing between the VoiceXML page 
and active objects. However, this has architectural imphcations that go beyond the scope 

M 

y of this application. Components were activated sequentially or in parallel We did not 

develop a firamework that enables object -oriented composition of the components. This 
lil 10 is just a limitation of the FDM as currently hnplemented; again not a limitation of the 
f firamework. 

yj Our implementation demonstrated the sufficiency of the framework to support 

p parallel activation of dialog components. On the other hand, we did not really confirm the 

usefiilness for application developers of a limited set of re-entrant reusable component as 
15 in the W3C reusable dialog requirement. Indeed, the use of transaction level re-entrant 
reusable components quickly appeared more efficient. This may just be because we did 
not allow building the transaction-level components from the foundation set in the W3C 
requirements. Because of the commonalties between objects and subdialogs, we consider 
that this also validated our requirements on VoiceXML subdialogs. 
20 IV, Implementation Smnmary 
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With respect to the VoiceXML specifications and dialog reusable components as 
described herein, it is preferred that support for client-side dynamic grammar compilation 
be added to the VoiceXML specification. Further, it is preferred that any reusable dialog 
component framework follows scopes as for subdialogs. In addition, it is preferable to 
5 limit the use of binary objects, i.e. non VoiceXML declarative, to the implementation of 
functions or behaviors not supported by VoiceXML. In addition, dialog modules that 
only use fimctions supported by VoiceXML should implement the reusable VoiceXML 
g dialog component and module framework, or its server side bean version, as described 

y herein. MoreovCT, a mechanian is preferably provided for sharing and aggregating 

WJ 10 parent form grammar with its subdialogs. Block behavior under mixed initiative should 
also be specified. 

Rl 

13 With respect to fiiture extensions of VoiceXML, service objects are preferably 

r^f implemented in addition to interaction objects. Such service object should enable 

dynamic data source access mechanisms (via HTTP (Hypertext Transfer Protocol) servers 
15 or by ODBC (Open Database Connectivity), SQL (Structured Query Language), etc). 
Further, objects and subdialogs are preferably re-entrant or support a re-entrant behavior 
and interfaces. Moreover, extensions of the VoiceXML interpreter and Form 
Interpretation Algorithm (FIA) are provided to handle modified or partially filled result 
subdialogs and objects and support mixed initiative across subdialogs, objects, documents 
20 and modaUties. 
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Although illustrative embodiments have been descaibed herein with reference to 
the accompanying drawings, it is to be understood that the present system and method is 
not limited to those precise embodiments, and that various other changes and 
modifications may be affected therein by one skilled in the art without departing &om the 
scope or spirit of the invention. All such changes and modifications are intended to be 
included within the scope of the invention as defined by the appended claims. 
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Appendix A: simpDate.gram 



gr aimaar com • ibm . speech . vxml . gr asnnar s_en . date ; 

// Copyright (c) 2000 IBM Corp. All Rights Reserved. 

// The return value is a string, in the format yyyymmdd, yyyymm??, yyyy????/ 
????mmdd, or ????mm??, 

// The grammar will return a in the appropriate positions if the user omits 
either the year or day. 

public <date> = <dayofmonth> <of_yr> <yr> 

{ 

this.$value = $yr + $dayofmonth; 
this.$tts = $dayofmonth.$tts + " + $yr,$tts; 

I } 

<dayo f mon th> 

{ 

this.Svalue = «????" + $dayofmonth; 
this.$tts ■ $dayofiiionth.$tt8; 
} 

I 

<mo> <of_yr> <yr> 

{ 

this.$value = $yr + $mo + "??"; 
this.$tts = $mo.$tts + " " + $yr.$tts; 

) 

I 

<yr_of> <yr> 

{ 

this.$value m $yr * "T???"; 
thls.$tts - $rr.$ttS7 

} 

I 

<mo_of> <mo> 
{ 

this.$value = "????" + Smo + "??"; 
this.$tts » $mo.$tts; 

} 

I <other> 

{ 

this.$tts = $other.$tts; 

} 



<dayofmonth> = ( <month29> [the] <ordinal01to29> | [the] <ordinal01to29> [day] 
of <month29> ) 

{ 

this. $values$inonth29 + $ordinal01to29; 
this.$tts»$month29.$tts + " " + $ordinal01to29. $tts; 

} 
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I ( <inonth30> [the] <ordinal01to30> | [the] <ordinal01to30> [day] 
of <]nonth30> ) 

{ 

this.$value=$month30 + $ordinal01to30; 
^ this,Stts=$month30.$tts + « « + Sordinal01to30, $tts; 

I ( <month31> [the] < ordinal 01 to31> | [the] <ordinal01to31> [day] 
o£ <month31> ) 



this. $ values SmonthSl + $ordinal01to31; 
this.$tts=Sinonth31.$tts + " " + $ordiiial01to31. $tts; 



<nionth29> = february {this,$value=n02«; this. $tts=" February"}; 

<month30> = april {this .$ valuer "04" ; this . $tts="April" ; } 

I june {this.$value="06"; this . $tts=:" June" ; } 

I September {tliis. $ values »»09"; this . $ttsa" September"; } 

I novenber {this. Svalue="ll"; this. St ts=" November"; }; 

<month31> = january {this. $value="01"; this. $tts=" January"; } 

I march {this. Svalue="03"; this. $tts= "March" ; } 

I may {this . $value="05" ; this . $tts= "May" ; } 

I july {this.$value="07"; this . $tts=" July" ; } 

I august {this.$value="08"; this . $tts= "August" ; } 

I October {this.$value="10"; this. $tts= "October " ; } 

I december {this. $valueB"12" ; this. $tts»"Deceniber"; }; 

<yr> = < teens > hundred 

{ this. $ value = $ teens. $ value + "00" ; 
this.$tts = $ teens. $value + "00"; 

} 

I <teens> ( <NULL> | hundred [and] ) <2dNumlto99> 

{ 

this.$value = S teens. $value + S2dNumlto99; 
this.$tts = $teens.$value + $2dNumlto99 . $value; 

} 

I ninety <numberslto9> 

{ 

this.$value = "199" + $mnnberslto9; 
this.$tts = "199" + $nuniberslto9.$value; 

} 

I two thousand [and] <2dNumlto99> 
{ 

this.$value = "20" + $2dNuialto99; 
this.$tts = "20" + $2dNuzalto99.$value; 

} 

I two thousand (zero | oh | nil] null | ought) <nuinberslto9> 

{ this.$value = "20" + "0" + $numberslto9; 

this.$tts = "20" + "0" + $ntmiberslto9.$value; 

} 

I two thousand 
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{ 

this.$value = "2000"; 
this.$tts s "2000"; 

I <2dNuinXto99> <nuinbers01to99> 
{ 

this.$value = $2dNiimlto99 + $iiumbers01to99 ; 
^ this.$tts = S2dNuinlto99.$value + $niiinbers01to99.$value; 

// I twenty <nuiiibers01to99> 

// { 

// this.Svalue « "20" + $numbers01to99; 

this.$tts = "20" + $nuinbers01to99.$value; 

// } 
; 

<ino> « <month29> { this.$tts = $nionth29. $tts } 

I <moiith30> { this.$tts = $inonth30.$tts } 

I <month31> { this.$tts = $month31.§tts } ; 

<other> = <NTn:iL> 

{ var months = new Array ("January" , "February", "March", 
"April", "May", "June", 

"July", "August", "September", "October" 

"November", "December") ; 

var days = new Array ("First" , "Second", "Third", "Fourth", 
"Fifth", "Sixth", "Seventh", "Eighth", "Ninth", 

"Tenth", "Eleventh", "Twelfth", 
"Thirteenth", "Fourteenth", "Fifteenth", "Sixteenth", "Seventeenth", 
"Eighteenth" , "Nineteenth" , 

■Twentieth", "Twenty First", "Twenty 
Second", "Twenty Third", "Twenty Fourth", "Twenty Fifth", "Twenty Sixth", 
"Twenty Seventh", "Twenty Eighth", "Twenty Ninth", 

"Thirtieth", "Thirty First") ; 
var daysInMonth = new 
Array (31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31) ; 

var currtime = new DateO; 

var yr = currtime. getFullYear () ; 
var mo = currtime. getMonth () +1; 
var da = currtime. getDate () ; 

if ((yr % 4) == 0) 

daysInMonth [1] ^ 29; 

} 



( today 

{ 



da. toStringO ) 
mo . toString ( ) ) , 



var da str = new String((da < 10) ? ("0" + da. toString () ) 

var mo_str = new String ((mo < 10) ? ("0" + mo . toString () ) 

this •$ value » yr + mo_str + da__str; 
^ this.$tts = months [mo- 1] + » ■ + days[da-l] + «, « + yr; 



yesterday 

{ 
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daml . toStrlng () ) 
mo. toString (} ) ; 



} 



yr = ((mo == 1) && (da ===1)) ? yr-1 : yr; 
mo = (da 1) ? (mo-1) j mo; 
if (mo 0) mo s 12; 

var daml = (da == 1) ? (daysInMonthEmo-1] ) : (da-1) ; 
var damlstr = new String ((daml < 10) ? ("0" + 
daml , toString 0 ) ; 
var mo^str = new String((mo < 10) ? ("O" + mo. toString () ) ; 

this.$value = yr + mo_str + daml_str; 

this.Stts = months [mo-ll + " " + daysEdaml-1] + " + yr; 



tomorrow 

{ 



dapl. toString 0 ) 
mo. toString 0 ) ; 

); 



yr = ((mo == 12) && (da == 31)) ? yr+1 : yr; 
var moO = mo; 

mo = (da == daysInMonthCmo-1] ) ? (mo+1) : mo; 
if (mo==13) mo = 1; 

var dapl = (da == daysInMonth [moO-l] ) ? 1 : (da+l) ; 
var dapl_str = new String ((dapl < 10) ? ("0" + 
dapl . toString ( ) ) ; 
var mostr = new String ((mo < 10) ? ("0" + mo. toString () ) 



this.$value « yr + mo_str + dapl_str; 
this.$tts B months [mo-1] + " " + days [dapl- 13 + 



■ + yr; 



// [in] [the year] [of] -all combinations except; "in of" 

<of_yr> = { in I in the year | in the year of I the year of I the year 

<NULL> ); 



of I 



<mo_of> s (in I in the month | in the month of I the month of I the month I 
<NULL> ); ' 
<yr_of> s (in I in the year | in the year of | the year of | the year | <NULL> ); 

<ordinal2to9> = second { this . $valu€="2" ; this. $tts=" Second" ; } 

{ this.SvaluesnS"; this . $ttSa:"Third" ; } 

{ this.$value="4"; this. $tts=" Fourth"; ) 

{ this.$value="5"; this . $ttSa«Fif th" ; } 

{ this.$valuew"6"; this. $tts= "Sixth"; } 
I seventh { this.$value="7"; this. $tts=" Seventh"; } 

I eighth { this . $value= « 8 " ; this. $tts« "Eighth" ; } 

j ninth { this. $value="9" ; this. $tts=" Ninth" ; }; 

<ordinallto9> = first { this, $value="l"; this. $tts= "First " ; } 

I <ordinal2to9> { this. $value=$ordinal2to9; 
this.$ttssi$ordinal2to9.$tts; }; 



= second 
I third 
I fourth 
I fifth 
sixth 



< ordinal lOtol 9 > 



tenth { this 
eleventh 
twelfth 
thirteenth 
fourteenth 
fifteenth 
sixteenth 
seventeenth 
eighteenth 
nineteenth 



} 



$ value= " 10 " ; thi s . $ 1 1 s = " Tenth" ; } 
this.$value="ll"; this. $tts= "Eleventh"; } 
this.$value="12"; this . $tts=s "Twelfth"; } 
this . $value=!"13" ; this . $tts= "Thirteenth" ; 
this.$value«"14"; this. $tts= "Fourteenth" ; } 
this.$value=:"15"; this. $tts= "Fifteenth"; } 
this.$valuea"16"; this. $tts= "Sixteenth" ; } 
this.$value="17"; this . $tts=" Seventeenth" ; } 
this.$value="18"; this . $tts="Eighteenthn ; } 
this,$value="19"; this. $tts= "Nineteenth" ; }; 



YOR9-2001-0463US (8728-525) 



-63- 



<ordinal2tol9> = <ordinal2to9> { this. $ valuer $ordinal2 to Se- 
this . $tts=$ordinal2to9 . $tts; } 

I <ordinall0tol9> { this. Svalue«$ordinall0tol9; 
this . $ttss$ordinall0tol9 . $tts ; } ; 

<ordinal02tol9> = <ordinal2to9> { this. $value="0" + $ordinal2to9; 
this.$tts=$ordinal2to9.$tts; } 

I <ordinall0tol9> { this. Svalue=$ordinall0tol9; 
this . $tts=$ordinall0tol9 . $tts; }; 

<ordinal01tol9> = first { this . $value="01" ; this. $tts= "First"; } 

I <ordinal02tol9> { this.$value=$ordinal02tol9; 
this . $ttsa$ordinal02 tol9 . $tts ; } ; 

<ordinaltens20to90> = twentieth { this . $value«"20 " ; this . $tts= "Twentieth" ; } 

thirtieth { this. $value=»30" ; this. $tts=: "Thirtieth"; } 
fortieth { this.$value="40"; this. $tts= "Fortieth"; } 
fiftieth { this.$value="50"; this . $tts="Fif tieth" ; } 
sixtieth { this. $value="60"; this. $tts=" Sixtieth"; } 
seventieth { this.$value="70"; this. $tts=« Seventieth" ; } 
eightieth { this. $value="80" ; this. $tts= "Eightieth" ; } 
ninetieth { this. $ values "90"; this. $tts= "Ninetieth" ; }; 

<ordinal2tol00> = <ordinal2tol9> 

<ordinaltens20to90> { this. $value=$ordinaltens20to90; } 

<tens> <ordinallto9> { this. $ valuers tens + $ordinallto9; } 

[one] hundredth { this. $value=" 100"; }; 

<ordinalltolOO> = first { this. $ valuer "01"; this- $tts="First"; } 

I <ordinal2tol00> { this . $valuea$ordinal2tol00; }; 

<ordinal01to29> = <ordinal01tol9> { this . $value=$ordinal01tol9; 

this.$tts=$ordinal01tol9.$tts; } 

I twentieth { this.$value="20"; 

this. $tts= "Twentieth" ; } 

I twenty <ordinallto9> { this. $value="2" + $ordinallto9 ; 
this. $tts= "Twenty " + $ordinallto9. $tts; }; 

<ordinal01to30> = <ordinal01to29> { this. $value=$ordinal01to29; 
this.$tts=$ordinal01to29.$tts; } 

I thirtieth { this. $value="30" ; this . $tts=" Thirtieth"; 

}? 

<ordinal01to31> = <ordinal01to30> { this. $ valuer $ordinal 01 to30; 
this . $ 1 1 SK $ordinal 0 1 to3 0 . $ t ts ; } 

I thirty first { this . $value="31" ; this. $tts«" Thirty 

First"; }; 

// The following were moved from the niunber grammar 

// (some of these are duplicated in the time grammar - if you change one here, 
change the other as well) : 

<2dNumlto99> = <tens20to90> <numberslto9> { this. $ values $ tens 2 0to90 + 
$numberslto9; } 

I <tens> { this. $value=$ tens + "0"; } 

I < teens > 

I <numberslto9> { this . $value=«0" + $numberslto9; 
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<nuiDib6rs01to99> ^ <nuinbers02to99> 

I (zero I oh I nil I null [ought) one { this. Svalue="01« i }; 

<nxainbers02to99> = (zero | oh | nil [null | ought) <nuinbers2to9> { this. $valueBno" + 
$numbers2to9; } 

I < teens > 

I <tens> { this.$value=$tens + "0"; } 

j <tens> <nuniberslto9> { this. $valueB$ tens + $nu]iibersXto9 ; 



<nu]nberslto9> 



= one { this. $ values "1"; } 

I <nuinbers2to9> ; 



<nuinbers2to9> s two 

I three 
I four 
1 five 
I six 
I seven 
I eight 
nine 



< teens > 



<tens> 



<tens20to90> 



{ this. $ values "2"; } 
this . $value= " 3 " ; 
this . $ values " 4 " ; 
this . $value="5" ; 
this,$value="6"; 
this . $ values "7 " ; 
this.$value="8"; 
this . $values »» 9 " ; 



B eleven 

I twelve 

I thirteen 

I fourteen 

I fifteen 

I sixteen 

I seventeen 

I eighteen 

j nineteen 

s ten 

I twenty 

I thirty 

I forty 

I fifty 

I sixty 

I seventy 

j eighty 

I ninety 

twenty 

thirty 

forty 

fifty 

sixty 

seventy 

eighty 

ninety 



this, 
this, 
this, 
this, 
this . 
this, 
this, 
this, 
this. 



$valuea"ll"; 
$values"12« ; 
$value="13" ; 
$value= "14"; 
$value="15"; 
$value=«16"; 
$value="17"; 
$values"l8"; 
Svalues"l9"; 



this . $ value= " 1 " ; 
this . $value= " 2 " ; 
this . $value= " 3 " ; 
this . $value= " 4 " ; 
this.$value="5"; 
this . $ values " 6 " ; 
this.$value="7"; 
this.$values"8"; 
thi s . $ values " 9 " ; 

this . $values " 2 " ; 
this . $ values " 3 " ; 
this . $values " 4 " 
this.$values"5" 
this. Svalues^e™ 
this. $value="7" 
this . $ values " 8 " ; 
this . $ values « 9 " ; 



// end of rules that were moved from number. gram and possibly duplicated in 

time, gram 
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